Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IaC #362

Closed
wants to merge 34 commits into from
Closed

IaC #362

wants to merge 34 commits into from

Conversation

james-otten
Copy link
Collaborator

@james-otten james-otten commented Apr 28, 2024

  • Terraform for creating node VMs in proxmox
  • Helm chart for creating resources
  • Incomprehensible instructions for creating a new environment (still includes manual steps)

Overview

Github secrets to be added

  • DEV0_KEY
  • DEV0_KNOWN_HOSTS
  • DEV0_SSH_TARGET
  • DEV0_PROJECT_PATH

TODO

  • Add github secrets (above)
  • Add CD
  • Setup certs - Not something I can do
  • Zero checkov findings

infra/helm/meshdb/secret.values.yaml Fixed Show resolved Hide resolved
.github/workflows/helm_lint.yaml Fixed Show fixed Hide fixed
@james-otten james-otten marked this pull request as ready for review April 28, 2024 20:27
@james-otten
Copy link
Collaborator Author

I think this is ready for initial review. dev0 isn't "done done" due to TLS, but these changes are additive and I'd like to test CD.

infra/README.md Outdated Show resolved Hide resolved
infra/README.md Outdated Show resolved Hide resolved
infra/README.md Outdated
Comment on lines 9 to 10
terraform plan --var-file=your.tfvars
terraform apply --var-file=your.tfvars
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we gotta fill out the example.tfvars?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to fill out the values in whatever var file you reference

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can that be added/explained in the docs?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, updated step 2

infra/setup_node.sh Outdated Show resolved Hide resolved
Comment on lines 23 to 26
# Setup secret files (will need to be modified)
cp meshdb/infra/helm/meshdb/secret.values.yaml ./secret.values.yaml
cp meshdb/infra/helm/meshdb/values.yaml ./values.yaml
cp meshdb/infra/tf/example.tfvars ./local.tfvars
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to call this out in the README as well

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one doesn't need to exist (the vm already exists)
The other ones are mentioned as ../../../../values.yaml and ../../../../secret.values.yaml

infra/README.md Outdated Show resolved Hide resolved
james-otten and others added 3 commits May 4, 2024 15:00
Co-authored-by: Willard Nilges <[email protected]>
Co-authored-by: Willard Nilges <[email protected]>
infra/README.md Outdated Show resolved Hide resolved
infra/README.md Outdated
terraform plan
terraform apply
# update address block in /opt/meshdb_mgmt/meshdb/infra/cluster/metallb_extra.yaml
kubectl apply -f /opt/meshdb_mgmt/meshdb/infra/cluster/metallb_extra.yaml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some instructions on how to get the kubeconfig might be good.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

scp -i tf/<private key> ubuntu@<control node>:/etc/rancher/k3s/k3s.yaml ./

@Andrew-Dickinson
Copy link
Member

Woah helm is insane. Y'all can run this show, I'll sit this one out

infra/README.md Outdated Show resolved Hide resolved
Copy link

@rossbannerman rossbannerman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! A couple of optional/nice to have things we might want to consider a bit further down the track:

  • Using resource type StatefulSet instead of Deployment for redis and postgres
  • Leaning on ingress-nginx (or another ingress controller) as opposed to a static nginx config
  • Having Terraform bootstrap the k3s cluster with ArgoCD and then have it manage rendering and applying charts/manifests

infra/tf/gen_ssh_key.sh Outdated Show resolved Hide resolved
Co-authored-by: Willard Nilges <[email protected]>
@WillNilges
Copy link
Collaborator

WillNilges commented May 11, 2024

Getting this when I try to run stage 2 again.

null_resource.mgr_stage_two: Provisioning with 'file'...
╷
│ Error: file provisioner error
│
│   with null_resource.mgr_stage_two,
│   on stage2config.tf line 9, in resource "null_resource" "mgr_stage_two":
│    9:   provisioner "file" {
│
│ Error connecting to SSH_AUTH_SOCK: dial unix /tmp/wilnil.agent: connect: no such file or directory

What exactly is the stage 2? It looks like it's setting up metallb and such? The stage2 script is running terraform... where exactly? I see a terraform zip on the k3s nodes... that seems a bit cursed. I have no idea if this is the kind of thing we should be doing with terraform.

Update: Looks like it wanted my ssh agent. When I start my ssh agent, it seems to be able to SSH, and I get this:

null_resource.mgr_stage_two: Provisioning with 'remote-exec'...
null_resource.mgr_stage_two (remote-exec): Connecting to remote host via SSH...
null_resource.mgr_stage_two (remote-exec):   Host: 10.70.90.230
null_resource.mgr_stage_two (remote-exec):   User: debian
null_resource.mgr_stage_two (remote-exec):   Password: false
null_resource.mgr_stage_two (remote-exec):   Private key: true
null_resource.mgr_stage_two (remote-exec):   Certificate: false
null_resource.mgr_stage_two (remote-exec):   SSH Agent: true
null_resource.mgr_stage_two (remote-exec):   Checking Host Key: false
null_resource.mgr_stage_two (remote-exec):   Target Platform: unix
null_resource.mgr_stage_two (remote-exec): Connected!
null_resource.mgr_stage_two: Still creating... [1m10s elapsed]
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 4: cd: meshdb: No such file or directory
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 5: git: command not found
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 6: cd: infra/cluster: No such file or directory
null_resource.mgr_stage_two: Still creating... [1m20s elapsed]
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 10: terraform: command not found
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 11: terraform: command not found
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 12: terraform: command not found
null_resource.mgr_stage_two (remote-exec): sed: can't read metallb_extra.yaml: No such file or directory
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 15: terraform: command not found
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 16: terraform: command not found
null_resource.mgr_stage_two (remote-exec): /home/debian/stage2.sh: line 17: terraform: command not found
null_resource.mgr_stage_two: Still creating... [1m30s elapsed]
null_resource.mgr_stage_two: Still creating... [1m40s elapsed]
null_resource.mgr_stage_two: Still creating... [1m50s elapsed]
null_resource.mgr_stage_two (remote-exec): error: the path "/opt/meshdb_mgmt/meshdb/infra/cluster/metallb_extra.yaml" does not exist
╷
│ Error: remote-exec provisioner error
│
│   with null_resource.mgr_stage_two,
│   on stage2config.tf line 14, in resource "null_resource" "mgr_stage_two":
│   14:   provisioner "remote-exec" {
│
│ error executing "/tmp/terraform_1068678211.sh": Process exited with status 1
╵

Confused as to why the SSH agent is necessary, but that aside, it seems like stage2 is expecting some stuff that isn't present. Could be due to me not having the agent set up when I was originally configuring.

Update

Got logs from a complete from-scratch run (which I think is really the way to do it). I think we gotta wait for the dpkg lock. Experimenting right now...

Interesting: null_resource.mgr_config_files (remote-exec): cp: cannot stat 'meshdb/infra/helm/meshdb/secret.values.yaml': No such file or directory. I guess that's because the branch is different. I wonder if we should just copy them over from the computer via terraform.

Gonna track stuff here: #368

Copy link
Collaborator

@WillNilges WillNilges left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one sec cooking

@WillNilges WillNilges self-requested a review May 14, 2024 23:37
@WillNilges WillNilges mentioned this pull request Jul 18, 2024
3 tasks
@WillNilges WillNilges closed this Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants